Azure AI Services at Scale for Cloud, Mobile, and Edge by Anand Raman & Chris Hoder & Simon Bisson & Mary Branscombe

Author:Anand Raman & Chris Hoder & Simon Bisson & Mary Branscombe [Anand Raman]
Language: eng
Format: epub, pdf
Publisher: O'Reilly Media, Inc.
Published: 2022-07-25T00:00:00+00:00

Using ONNX in ML container runtimes

Portable machine learning systems are an important tool for edge computing, putting ML models where they’re needed. To simplify deployment you can package an entire ONNX runtime in a container, along with your ONNX model. Deployment is simply a matter of loading and running the container on an edge Kubernetes system like K3S or using a standalone container runtime. Updates are handled by swapping out containers with minimal service downtime and disruption.

Setting up an ONNX runtime container is easy enough. Start by building a container image that’s configured with Python. Use pip to install the ONNX runtime, either for CPU or GPU. The Python ONNX runtime loads the model as an inference session, and you can use it to extract model data to enumerate inputs and outputs, as well as confirming the model metadata.

Running a model is as easy as calling your inference session with your model inputs. You can define which outputs you want to use or simply accept all responses.

Microsoft provides a prebuilt ONNX runtime container for use with pre-trained Azure ML models. Based on Ubuntu 18.04 and Python 3.7, it’s designed to use CPU inference and should run on most Docker or containerd systems. Preinstalled packages also include NumPy and pandas, so you can work with results using common numeric methods. The prebuilt dockerfile can be used as the foundation for your own image, adding additional libraries and code as a nested docker image.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.